SimHEC: Understanding Application Efficiency at Exascales through Simulations

It is expected that our HEC system will enter exascale era in decade, which is one thousand times of performance as today’s system (petascale). In the mean time, many challenges also have been noticed and pointed out, as the size of HEC system increased without some dispensable improving on architecture of today’s HEC system, the systems could collapse at exascale, because the functionality would not be able to complete their duties successfully and even break down the whole system. One potential problem is that the MTTF (mean time to failure) of a HEC system will decrease linearly as the system size increases. We will probably have to face the serious situation that our HEC system might be very unreliable and no works could be done successfully, due to too frequent failures. The situation could be worse for HEC with parallel file system, even armed with checkpointing to guarantee its reliability. In this project, we study and explore application efficiency toward exascale, and show that DFS offers a lot better performance by taking its advantage of the local storage speed. We measure the application efficiency in both parallel and distributed file system, scale them from a small system size to exascale with and without checkpointing, and observe the differences between three different workloads: uniform workload with only one job running at anytime, uniform workload with ten jobs running concurrently at anytime, and Intrepid system’s 8 month workload from Argonne National Laboratory. All the experiments were simulated in the Java simulator developed in the DataSys laboratory at Illinois Institute of Technology.